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COMMUNICATION WITH Mm.TI- SENSORY DEVICES 
Field of the Invention 



The present invention relates to multi -sensory devices and more 
particularly to communication with multi-sensory devices 

Background to the Invention 

The mobile telephone and/or other small portable devices (such as' 
PDAs) are now enabled for user access to, for example, the World Wide Web, 
driven by the Wireless Application Protocols (WAPs) . As a result there is 
now consumer expectation that such devices will enable access to services 
or applications, for example the internet, in much the same way as can be 
achieved with a desktop personal computer. 

Further, many of these devices are multi-sensory, primarily visual 
and aural, through allowing input and output interaction by use of a 
limited viewable area and a limited keyboard or a stylus with pointing and 
gesture capabilities, and by use of audio techniques (voice in or out). For 
example, consider a mobile telephone. The user may be looking at the 
viewable area and controlling the device through the keys when dialling, 
searching the address book, interacting with text messages, playing a game, 
or surfing the net. Similarly, the user may be using the mouthpiece and 
earpiece when conducting verbal communication. Also, in either case, the 
other sense may be participating in a secondary manner. For example, the 
user may be looking at the viewable area and also hearing information by 
way of a connected button earpiece. 

However, one difficulty associated with this type of use occurs when 
there are bandwidth capacity constraints for communications between a 
device and the server which provides the services or applications in use by 
the device. For example the server may have data of different types to send 
to the multi-sensory device which requires, for timely delivery, more 
bandwidth than is available on the communication channel. In this situation 
the server may send data to the device which, from the aspect of the user, 
does not make effective use of the bandwidth available, for example by 
sending video data when the device is only being used aurally. 

Note that US 2002/0081972 to Koninklijke Philips Electronics N.V. 
discloses adding an additional field to .-inqui-y messages sent from a mobile 
device, the additional field containing personal data of a user. The 
personal data provides information which relates to the user, such as 
gender, music tastes, sports interests, etc. and these are used to decide 
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Alternatively the data items further have an assigned priority value, 
and as a result rather than allocating priority the existing priorities are 
increased according to the usage mode. As a result the assigned priority 
value for video data items is increased if the usage mode indicates that 
the device is being used visually and the assigned priority value for audio 
data items is increased if the usage mode indicates that the device is 
being used aurally. This enables existing priorities of data items to be 
reflected in the priority order which results from the usage mode 
indication. 



In prioritising the sending of data in this way the data processing 
host can make more effective use of the communication bandwidth with the 
device. Note that prioritisation could involve sending primarily or 
exclusively data items which match the current usage mode of the device. 

Optionally the usage mode indication is also provided to one or more 
applications that are communicating with the device. For example an 
application could register to be informed of the usage mode. Alternatively 
an application could request the usage mode when needed. This enables an 
application to tailor the service it is providing to the device according 
to the mode in which the device is being used. 

Preferably the communication is intercepted in the communication 
protocol stack configured on the mobile device, for example the Wireless 
Access Protocol stack. In such a protocol stack the intercepting step 
preferably intercepts the communication in a protocol layer which 
implements any one of: Wireless Transaction Protocol; Transmission Control 
Protocol; and HyperText Transfer Protocol. 

According to a fourth aspect the present invention provides a data 
processing method for a multi- sensory device to communicate a usage mode to 
a server, the method comprising the steps: receiving a usage mode 
indication of the multi-sensory device; intercepting an outbound 
communication; and adding the usage mode indication to the outboiond 
communication . 



According to a fifth aspect the present invention provide a 
multi-sensory device for communicating a usage mode to a server, the device 
comprising: means for obtaining a usage mode indication of the 
multi-sensory device; means for intercepting an outbound communication; and 
means for adding the usage mode indication to the outbound communication. 

According to a sixth aspect the present invention provides a computer 
program product coit^rising instructions which, when run on a data 
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Figure 3 is a flow diagram of the method of a multi-sensory device 
for sending a usage mode indication to a data processing host; and 

Figure 4 is a flow diagram of a method of a data processing host when 
processing a usage mode indication received form a multi-sensory device. 

Description of the Preferred Embodiment: 

According to the preferred embodiment of the present invention a 
multi-sensory mobile device, such as a mobile phone or personal digital 
assistant (PDA), has an awareness of the mode in which it is being used 
(usage mode) and can convey such information to the searver (data processing 
host) with which it is communicating. For example, for a WAP phone, the 
usage mode would indicate whether the phone is being used to communicate 
aurally (e.g: for a phone call) or visually (e.g: for viewing an internet 
page) . Further when the usage mode changes the new type of use is conveyed 
to the server. 

The device conveys usage information to the server as part of a 
header of a protocol, for example the Transmission Control Protocol (TCP) 
header, which the device uses to communicate with the server and not as 
part of application data for an application running on the device, for 
example by the internet browser being used. Use of a protocol header, as 
opposed to application data, provides the advantage that the server may use 
this information for a plurality of applications in commiinication with the 
device and further use of the application data would be invidious (and 
potentially impossible if the application data were encrypted) for the 
server to inspect the application data destined for the application. 

Further, in the preferred embodiment the server with which the 
multi-sensory mobile device is in communication uses the usage mode 
information received from the device to decide how to most effectively 
communicate with that device. For example the server may use the usage mode 
information to optimise its communication in terms of the priority order it 
uses when deciding the type of data send to the device. 

The preferred embodiment of the present invention is illustrated in 
Figure 1 which is a schematic diagram of three mobile multi-sensory devices 
(101,102,103) conveying a usage mode indication (121,122,123) to a server 
(104) with which they are communicating. Multi-sensory device 101 is a 
mobile phone and is currently being used to hold a telephone conversation. 
As a result the device includes a usage mode indication 121 of aural" with 
outbound communications 111, which in this case comprises the speech of the 
user, to the server 104. Multi-sensory device 102 is also a mobile phone 
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that visual data items (211, 212, 213) be sent to the device. For example. 
Application 1 could be providing a clock which sends time data in the form 
of an analogue clock which is displayed in a window in some part of the 
screen of the device and is accessed at regular intervals. Application 2 is 
an audio/video application which has requested that different sorts of data 
items be sent to the device. These data items are audio data items 
(221,224), mixed audio and video data items (222) and video data items 
(223). For example, application 2 could be providing an audio/visual 
presentation with a mixture of silent images, movie clips (images with 
sound) , and audio which provides background information about silent images 
and/or video clips. For example the clock of application 1 could overlay 
Application 2 or be hidden by Application 2. Application 3 is an audio 
application and has requested just audio data items (231, 232) be sent to 
the device. For example application 3 could be a normal telephone 
conversation, a talking newspaper service, or a talking advertisement 
service. 

Note that, in fig. 2, the applications and the associated data items 
which they have requested be sent to the device are just examples. In 
practice the device may be running one or more applications, and an 
application may send one or more data items. 

Further note that the application data held in a transmission queue 
does not necessarily originate from applications running on the same server 
as the server on which the transmission queue is held. For example the 
server could be acting as a gateway between the multi- sensory device and an 
application. 

According to the preferred embodiment of the present invention the 
server receives an indication of the usage mode of a device and processes 
the transmission queue for the device, such as shown in figure 2, according 
to this usage mode. 

Figure 3 is a flow diagram of the method of a multi-sensory device 
for sending usage mode information to the server, according to the 
preferred embodiment of the present invention. At step 3 01 an indication of 
the usage mode is received from another component of the multi-sensory 
device. At step 302 an outbound commiinication of application data is 
intercepted. For example this can be done at various levels in a WAP stack, 
for example depending on the implementation of the WAP stack, at the 
Wixeiess Transaccion Protocol (WTP) , Transmission Control Protocol (TCP) , 
or HyperText Transfer Protocol (HTTP) layer. Finally at step 303 an 
indication of the usage mode is added to an appropriate protocol header of 
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the WAP stack and the layer will depend on the layer in which the usage 
mode indication was added by the multi- sensory device at step 303 of fig. 
3. Further, note that this communication could involve transmission of 
application data from the device to the server, for example voice data as 
part of a telephone conversation or a request for a service, or could be a 
communication to specifically pass the usage mode data to the server (i.e.: 
a communication with no application data) . 

At step 402 the usage mode is provided to any applications which have 
requested to be made aware of it. For example, a clock program, such as 
application 1 of fig. 2, could wish to know the usage mode if it provides 
an alarm function and depending on the usage mode of the device it sends a 
visual or audible alarm. 

At step 403 a check is made to see if the usage mode received from 
the multi-sensory device indicates ''aural", and if it does at step 404 the 
server tailors data sent to the device to favour audio data. For example, 
with reference to figure 2, the server could assign a higher sending 
priority to data from Application 3 and Application 2, compared to data 
from Application 1 which comprises only video data items (211,212,213). 
Further data from Application 3 which comprises purely audio data items 
(231, 232) may be given a higher sending priority than data from 
Application 2 which comprises audio data items (221,224), video data items 
(223) and mixed data items (222) . Further with regards to the application 
data from Application 2, the audio data items (221, 224) may be given a 
higher sending priority than the mixed data items (222), and the mixed data 
items (222) may be given a higher sending priority than the video data 
items (223) . Note that assigning data item a higher sending priority 
results in it being preferred for sending to the mobile device compared to 
lower priority data, thus setting a priority order. 

If the usage mode does not indicate ''aural", at step 405 a check is 
made to see if the usage mode indicates "visual", and if it does at step 
406 the server tailors data sent to the device to favour video data. For 
example with reference to figure 2 the server can give a higher sending 
priority to video data items from applications, such as Applications 1 and 
2, and further give a higher sending priority to video data items from 
applications which send only video data, such as Application 1. Further, 
for a given application, such as Application 2, the video data item (223) 
may be given a higher sending priority than the mixed data item (222), and 
mixed data item (222) may be given a higher sending priority than audio 
data item (221, 224) . 
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with reference to figure 3 and 4 could be embodied in a data processing 
apparatus . 

Thus the present invention provides a method, apparatus and computer 
program product which enables the mode of use of a multi-sensory device to 
be used to prioritise the type of data which is sent to the device. The 
multi-sensory device sends its usage mode, for example, ^aural" or 
^visual", as part of communications to a server. The server then obtains 
the usage mode and uses this information to prioritise data sent to the 
device. For example if the device is being used aurally, audio data is 
prioritised for sending to the device such that data sent to the device is 
primarily or exclusively audio data. 
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5. The method of any preceding claim comprising the further step of: 
providing the usage mode indication to one or more applications; 

6. The method of any preceding claim wherein the intercepting step 
intercepts a communication in a protocol layer which implements any one of 
Wireless Truncation Protocol; Transmission Control Protocol; and HyperText 
Transfer Protocol. 

7. A data processing method for a multi-sensory device to communicate a 
usage mode to a server, the method comprising the steps: 

receiving a usage mode indication of the multi-sensory device; 
intercepting an outbound commimication; and 

adding the usage mode indication to the outbound communication. 

8. The method of claim 7 further comprising the further step: 

generating the outbound communication which is intercepted by the 
intercepting step. 

9. The method of either claims 7 or claim 8 wherein the usage mode 
indication indicates that the device is being used either aurally or 
visually. 

10. The method of any one of claims 7 to 9 wherein the intercepting step 
intercepts a communication in a protocol layer which implements any one of: 
Wireless Truncation Protocol; Transmission Control Protocol; and HyperText 
Transfer Protocol. 

11. The method of any one of claims 7 to 10 wherein the multi-sensory 
device is a mobile phone. 

12 . A data processing apparatus for processing a usage mode indication 
received from a multi- sensory device, the apparatus comprising: 

means for intercepting an inbound communication from the device; 

means for obtaining a usage mode indication for the device from che 
communication; 
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means for obtaining a usage mode indication of the multi-sensory 
device; 

means for intercepting an outbound communication; and 

means for adding the usage mode indication to the outbound 
communication; 

18. The device claim 17 further comprising: 

means for generating the outbound communication which is intercepted 
by the intercepting means. 

19. The device of either claims 17 or claim 18 wherein the usage mode 
indication indicates that the device is being used either aurally or 
visually. 

20. The device of any one of claims 17 to 19 wherein the device is a 
mobile phone. 

21. A computer program product comprising instructions which, when run on 
a data processing host, cause said data processing host to carry out a 
method according to any one of claims 1 to 11 . 
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